Computing Iceberg Queries Eeciently Paper Number 234

Many applications compute aggregate functions (such as COUNT, SUM) over an attribute (or set of attributes) to nd aggregate values above some speci ed threshold. We call such queries iceberg queries because the number of above-threshold results is often very small (the tip of an iceberg), relative to the large amount of input data (the iceberg). Such iceberg queries are common in many applications, including data warehousing, information-retrieval, market basket analysis in data mining, clustering and copy detection. We propose e cient algorithms to evaluate iceberg queries using very little memory and signi cantly fewer passes over data, as compared to current techniques that use sorting or hashing. We present an experimental case study using over three gigabytes of Web data to illustrate the savings obtained by our algorithms.